Representing human uncertainty by subjective likelihood estimates

We give a definition of human uncertainty through subjective likelihood estimates. The subject is asked for his estimated likelihood of a discrete variable, given a present piece of uncertain observation, under the hypothetical assumption that the variable was uniformly distributed prior to the new observation. With this interpretation of human uncertainty, we are able to perform consistent inference about our target variable, by formally treating the input as likelihood factors. The algorithm has been successfully implemented in an expert system for classification of wildwood mushrooms. Introduction People frequently make statements like: “I’m 90% sure that the taxi driver spoke Swedish in his cell phone.” The purpose of the present article is to give a probabilistic interpretation of statements of this type, so that we can combine them, and produce consistent inference. A simple approach would be to claim that when a person makes the given statement, he is right 90% of the times. This makes some sense when there are only two alternatives, if we assume symmetry in his errors, so that his 10% error rate applies whether the driver actually speaks Swedish or not. However, we would like to generalize our interpretation to cases with more than two possible answers. Suppose our subject estimates the taxi driver’s language to be: “90% Swedish, 5% Norwegian, 3% Danish and 2% Icelandic” Then the error rate interpretation fails to make sense. The article is laid out as follows: First we review different established models of human uncertainty. Then we define our model of subjective likelihoods and give a Bayesian inference rule for combining statements. Then we describe an application of the algorithm in an expert system that helps a user classify wildwood mushrooms. The last section concludes the article. Established models of human uncertainty In this section we give a broad overview of models that have been used for quantifying human uncertainty. Certainty factors In the early days of artificial intelligence, expert systems were built that were imitating human inference (Shortliffe, 1976). The typical expert system consisted of a set of facts, a set of rules, and an inference engine. The inference engine applied a sequence of rules to the set of facts, thereby producing new facts. Uncertainty was modelled through certainty factors associated to facts and rules. Although some expert systems of this kind worked quite well, certainty factors are not popular nowadays, because they tend to produce contradictions. Fuzzy logic Fuzzy logic attempts to model uncertainty through vagueness, rather than probabilities. In a fuzzy logic context, our taxi driver example statement would be interpreted as: “On a swedishness scale from 0 to 100, the taxi driver’s language was 90.” This is an interesting and useful semantic model in many cases, but our goal is to model the fact that the subject’s observation may be wrong, not that he is correct to a certain degree. For a discussion on how fuzzy logic relates to probability theory, see (Dubois & Prade, 1997). More fundamental connections between the theories of standard and fuzzy sets are made in (Indahl, 2000). Dempster-Schaefer theory In Dempster-Schafer theory, uncertainty is modelled by an interval ( ) [ ] , 0,1 a b ⊂ (Shafer, 1976). The idea is that the span of the interval reflects the degree of uncertainty. One might e.g. assign [0,1] to a statement of extra terrestrial intelligent life, while the event that the future flipping of a fair coin gives “heads”, would have a collapsed interval {0.5}. The theory gives a consistent calculus for combining statements. It is not readily applicable to our setting, though, because our subject does not convey his uncertainty in the form of intervals. Lower previsions The theory of lower (and upper) previsions can be seen as a generalization of D-S theory (Walley, 1996). The lower prevision of a statement can be interpreted as a lower limit of the probability of the statement being true. The theory is related to gambling situations where one assumes that the opponent may have more information than oneself. As an example, you might assign a 0.4 lower prevision on the event that the flipping of a coin gives “heads”, if you suspect that the coin may be unfair, but you are sure that even an unfair coin will give heads at least 40% of the time. The theory is by nature pessimistic, as it always works through worst-case values of probabilities. This is good for the purpose of making robust inference, but does not capture the meaning of out taxi driver example statement. Subjective Probability A natural interpretation of our example statement is that the subject’s subjective probability of the driver’s conversation being in Swedish is 0.9. The term subjective probability (as opposed to frequency based probability) means that the subject merely assigns numbers to different events and statements, which obey the rules of probability calculus. A problem with subjective probabilities is that one cannot easily combine different subjective probability statements in a meaningful way, because the statement is derived from the subject’s internal probability model. Suppose we want to combine the given statement with the fact that the event took place in Sweden, we would first need to know whether the subject had already included this important piece of information in his 0.9 probability estimate. Also, it is very hard for people to produce consistent subjective probabilities in cases where they simply do not know. It is well known that the attempt of assigning uniform probabilities to reflect ignorance fails. The question of extra terrestrial intelligent life is a good example: If you assign a 0.5 probability of extra terrestrial intelligent life in our galaxy, you cannot readily assign the same probability for the left arm of the galaxy, or for entire universe. The difficulty in representing ignorance is a big problem with subjective probability models. Bayesian networks In a Bayesian network (Jensen, 1996), nodes represent random variables, which are connected through edges that represent causal relations. When new evidence is presented, probabilities are propagated through the network in a consistent way. Bayesian networks represent a different perspective than that of classical expert systems: Rather than imitating the human thought process, with uncertainty associated to inference rules, one creates a consistent causal probability model, and uses probability calculus for inference. Under this paradigm, certain and uncertain human knowledge is included in the model of the world, rather than in the automatic reasoning. Hence, Bayesian network modelling does not offer any immediate interpretation of our taxi driver statement, but it gives a framework within which we would like our interpretation to fit. Subjective likelihood Our interpretation of the taxi-driver statement, which we introduce in this article, is this: “The probability of me hearing what I heard, if he did speak Swedish, is nine times higher than the probability of me hearing what I hear if he didn’t speak Swedish.” With this interpretation, the statement only refers to the present observation, not the subject’s overall judgment concerning the driver’s language. By only referring to the subject’s present observation, and not to his personal beliefs about the probability of meeting Swedish-speaking people in this given situation, the statement is made context free. This enables us to use it in a formal probabilistic Bayesian model, and combining it with other statements, without worrying about the statement’s context. Now assume that the subject is in Sweden, where the a priori probability of a taxi driver speaking Swedish on the phone is, say, 95%. Then the likelihood of the conversation having been in Swedish is the prior probability of 0.95 multiplied by the observation weight 0.9, while the likelihood of the opposite is 0.05 times 0.1. This gives: [ ] 0.95 0.9 Conversation in Swedish 0.994 0.95 0.9 0.05 0.1 P ⋅ = ≈ ⋅ + ⋅ This high estimate is reasonable, because the conversation both sounded Swedish to the subject, and took place in Sweden. We formalize this calculation for observations with n different values. Let { } 1 2 , ,..., n θ θ θ θ ∈ Θ = be the true state of Nature, and let the prior distribution p be a probability vector of length n, so that ( ) i i p P θ = . Let { } 1 2 , ,..., n o o o be a vector of random variables with values in some space X. We interpret X as the set of possible observations that the subject can make, and the random variable i o represents the random observation that the subject makes, given i θ θ = . Assume now that the subject has made observation x. We can then calculate the conditional distribution ( ) | p x using Bayes formula: ( ) 1 1 ( ) ( | ) ( ) | ( | ) ( ) ( | ) ( ) i i i i i n n i j j j j j j P P x p P o x p x P x P P x p P o x θ θ θ θ θ = = = = = = = ∑ ∑ (1) In this formula, we use the standard statistical convention of interpreting ( ) i P o x = as a probability if the i o ’s are discrete variables, and as probability density (likelihood) otherwise. (In theory, one should link this to the structure of the observation space X, but we will return to this below.) So far, our construction is one of standard Bayesian inference. The next step in an applied Bayesian analysis would usually be to collect data (x), and compute ( ) | p x , treating the distributions of the i o ’s as given. Our approach is simpler mathematically, as we leave the assessment of o, x and X to the subject. We define the subjective likelihood vector of observation x X ∈ by: [ ] 1 2 ( ), ( ),..., ( ) n q P o x P o x P o x = = = = (2) Again, we either treat ( ) i P o x = as a probability or a probability density. Note that the observation x and the random variables i o are “private” for the subject, which is the reason why we can disregard the mathematical structure of the domain X. For our purpose, a rescaling of the vector q is also of no importance, as only the components’ relative values affect our computation below. In a sense, we condition p by the vector q (which is what our subject reports), so write ( | ) p q instead of ( | ) p x . This is a slightly abuse of notation, but we prefer to hide the private variable x. With our definitions, equation (1) now simplifies to: